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.ABSTRACT 

^A study was conducted to measure the faciiiity of kindergarten pupiis in 
taking an FYCSP Criterion Exercise on optically scannable sheets. Three 
types of response boxes were tested. .It was concluded that .pupils are. 
able to ha^adle a ten-i^ with xniiiimal practice and 

directions. 
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PRIiUMINARY TRYOUT OF OraCALLY SCANNABLE FYCSP CRITERION EXERCISES 

Introduction _ " . - 

Little attention has been paid to the ability of kindergarten 
.children to us'e optically scannable response sheets in an instruc- 
tional management setting. Pupils participating in the SWRL FYCSP 
presently use a one-sheet, four-page set of multiple choice items, 
selecting their answer by , placing an "X" in the box under the appro- 
priate answer. A machine readable Criterion. Exercise is necessary 
to permit source, data automation of an: instructional management 
system. 

Therefore, three response formats were designed for pre^IMS- 
instalTation tryqiit. The forms contained^ ten responses^ per side, 
three; response choices per item, a picture for each= numbered response, 
and a page number . Additional f eatures^ included^ preceding of program 
name, unit number ,. and page nu#er, as well as space for a binary - 
pupil code. Thus, while the content is the same as the EYCSP Crite- 
■ tion Exercise, the ^hree forms differ in appearance, the main differ- 
ence being ten items per sicie versus five. 

Two schools, a total of five kindergarten classes, in the 
tos Angeles City SchpoTs, were selected for work in the tryout. Whrle 
the :purpose of the tryout was manifold, a basic intention was to ^ 
detefrniiie rf pupils below grade one can properly fill in a optically 
scannable Criterion Exercise form. Psychometric riteratufe suggests 
that grade three is the minimal lowest grade level tu introduce these 
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£om)s, but the IMS situation is suf|ic:iently different that feasibil- 
Llry at the kindergarten level was felt to be an attainable, objedtlve. 

Background and Related Literature 

Gaffney and Maguire (1971) conclude that pupils in grades four 
and five can handle separate mark- sense response sheets when provided 
with specific directions and practice. Pupils above fifth grade were 
able to make valid responses regardless of type of instruction, 
Second-and third grade pupils were unable to use separate answer^/ 
sheets with success. All pupils involved;, lie,, grades'^ two through 
eight, were able to answer questions effectively when using a test 
booklet that was not machine scorable. Cashen and Ramseyer 01969) 
also tested pupil ability use separate answer sheets in virtually 
the same manner as the above study. Results Indicate that third 
'grade pupils above average a>tllty used the sheets effect ively^^^ 
McKee (19670 , although presenting no empirical data, conclude that 
third'. grade pupils "seemed to" be aFle to use a separate answer sheet, 

Culhane and Stodola (1967) administered: opi^^^^^^ questions to 
pupils in grades one through eight and found: that all were able to 
respond on mark- sense IBM cards, even though many of the marks were 
too light to be read by machine. The dependent variable in this study 
was the mumber of marks the machine could read in a designated field, 
whtle in the first twp studies, correct response^^ on a ;paraMel form 
test were the criterion. Hieronyroous (196^)., working with third 
graders, found no significant difference among three training methods,. 
However, alrt puptls were able to use separate answer sheets effectively.. 



From these studies it was concluded that: 

1. separate answer sheets can be used on achievement tests with 
third grade pupils; 

2. Training pupils on^ the use of separate sheets appears impor- 
tant^ but no one method has been shoxm to be significantly 
more effective than others; 

3. Hark- sense cards were used as early as first grade to gather \ 
. data on ah opinion questionnaire; 

4. No work with kindergarten pupils on scannable answer sheets 

has been reported. 
Ml J Statement of the Problem 

It becomes obvious that^hr^acility. of kindergarten^p in 
':handring\ scannable response sheets is an empirical question. The 
problem to be resolved is presented in-fbur parts: ' ^ 

1. Gan kindergarten pupils respond to oral questions on a; mark- 
sense Criterion Exercise? . 

2. How does fot^n :design af^ect pupil marking behavior? 

3. mat training materials and zprocedures elicit appropriate 
pupil response? 

* 4.. How effectively does the SWU. optical scanner read kindergarten 
pupil response sheets? ' ^ 
The problem of quality control on STOL production: of Criterion 
Exercises has been resolved by Gibbs and Hooper (^N: 5-72-05:).. It was 
determined that Production Services could produce forms with .print 
tolerances, paper stock specifications, and ink dharaQterist 'cs that wou^ 
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meet requirements set forth by the Optical Scanning Corporation. What 
remained to be answered was the question of grade level, forms design, 
and training appropriate for development and use of machine readable 
Criterion Exercises.- ' 

Procedures 

^A-.r — Matferials^ " ' " 

Criterion Exercises for FYCSP Unit 1 were generated. The tests 
comprised one sheet, two sides, ten items per side, with a picture 
as well as a number for each item to assist pupils in identifying, 
the proper response row.. Three-distractofs per item ye re presented, 
with a response box below each option. Pupils- vere instructed to 
fill in-the box under the correct response, making sure that the 
mark was dark and shiny.. (Regular EYCSP dif actions instruct 
pupils to pldce an"X" In the appropriate box. ^ Three different 
shapes of boxes were used: vertical, horizontal, and square. The 
exercises were alike in all other aspects. 

Gharacterlstics of tryout materials and regular Criterion Exercises 
are compared in Figure 1, 

B. Population 

Seventy- six pupils from five kindergarten classes, in- two schools 
were involved. The age range was approximately four-and-one-half 
years to six -years. SWRL requested that all pupils involved be 
using FYCSP. However,,, at le.ast three classrooms, were still 
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using the Instructional Concepts Program and had no,t begun the 
Communication Skills Program. 



///sheets' 

size of sheet 

#/pages 

#/ items/ page 

paper stock - 

ink 



FYCSP- regular 
1 

11" X 17" 
4 
5 

refined newsprint 
40 ?pbund book 

black 



FYCSP-tryout (scannable) 

8V' 3C 11" 

-2 

10 - ' 

60 :p6und offset white 
Mustang,^with Vellum f inish 

black plus drop-put blue 
for marking boxes; 



Figure 1. Comparison of tryout raaterials with: Criteridh^ 
Data 

Data comprised marks made by pupils on Criterion Exercise response 
sheets. Darkness of mark, direction of pencil stroke, multiple , 
missing,, and: crosse:d out responses, as well as stray marks- were 
counted. Amount of test taking time and observations on :teacher 
and: puptl behavior were also recorded,. 

AiialysTs 

Three types of analyses were conducted: 1) an examination of 
the manner in. which response positions were marked. This included 
:the pencil direction used to mark the response, the number of 
multiple marks , stray and missing marks , and changes In response , 
choice; 2) an analysis of the Digitek DM/ iOO scanner's ability 
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to read the pupil marks entered in the response positions; and 
3) notation of pupil and teacher behavior during test administra- 
tion, plus SWRL observer reactions. 

Table's 1-3 present an analysis of marks made on the Criterion • 
Exercises. ■ The percentage figures for all measures other than- 
stray mark^ refer to item (Item N.= 20 for all forms).. Stray 
marks are reported by average number per test sheet. For each 
exercise sheet,, there was approximately one stray mark, which 
would be read by the scanner. Such marks must be erased by 
clerical' help =bef ore the exercises are read . The iperceritage of 

. total itemsi that contained ^Itipie.marks^^^^ 
for the square= boxes to 3.1: perce>t for the v#tic al . Hor izdnt al 

■ marking boxes produced; a total of 17 unm:arked responses , or 
3,4 percent of atl items. The other two forms ^produced a mich 
lower percentage. The vertical box exercises shewed a totat of 
16 rei5pbnse changes Oindicated- by crossing but the utiwanteji 
response). 

Table 4 presents a comparison of olcier (five years and above) 
versus younger (belpw ;f ive years) puptls in the one class, using 
-the horizontal marking boxes. Younger pupils (under five- years) 
made- more stray marks , raurtipTe marks, mark changes, and left 
more items unanswered. , ! . 
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Table 1. Analysis of pupil marks when vertical response boxes 
were used. Pupil N = 36. 



DESCRIPTOR 




N 


— a 


Stray Marks 




76 
22 


1.06 

3 .1 


llultiple Marks 




No Mark 




7 


i!o - ;: 


Mark Changed 




, 16 


2.2 


^ Average per side 








Table 2i Analysis of pupil 
boxes were used. 


taarks=^when horizontal response 
Pupil N = 25. 


DESCRIFTOR 




N 


- / — a 

^ / ■ . ^ ^ = ^ 


Stray ^Marks 




61 


-4 1.02 


Multiple Marks 




24 


4.8 ■ - 


No Mark 




17 


3.^ ' - 


Mark crK'inj;ud 




. 3 


. 0.6 - ■ 


Table 3. Analysis of 
were useci. 


pupil- fnarks when square response boxes 
Pupil N = 15. 


DESCRIPTOR 




! N 




I ; stray Marks 




5 . 


- ■ .17 


Multiple Marks 




19= 


6.3 - i 


No -Mark- 




2 




Mark- Changed 




1 


.3 
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Table 4. Analysis of pupil marks when horizontal boxes were 
used (older versus younger pupils). 



DESCRIPTOR 



GROUP 



— a 



Stray Marks 



Multipie'tMarks 



No Mark, 



Mark Changed 



Younger 

Older 

Total 

Younger 
Older 
Total- 
Younger 
Cider 
Total 

Younger 

OWerl 

Total' 



43 
18 
61 

19 
5 

24 

17 
6 
17 

3 
0 
3 



1.95 
0.64 
1.02 



8.6 
1.7 
4.8 

7.7 
0.0 

i-.3 

0.0 
0.6 



An effort was mtde to= detennine how fupils fitled in the response 
boxes . The tejcher s were instructed to-tell their pupils to f ±^11 
th6 boxes In compi&iy. Tnakini. sure that the marks were dark andi 
shiny. They were not ;told the manner in which they were t^o in^ke 
their mark. Table 5 shows the number of boxes filled in by marks 
judged to-be vertical (U) , horizontal (H) , circular <(C) , or other 
(0) . "Other" included combinations of the first three , ibasicali;y> 
When presented; with vertical boxes , puptls marked: 93 percent of 
the response positions wi^h vertical marks. Sixty- seven percent 
of tie hpr izont al boxes were marked: in a hor izont al manner.. When 
square boxes were used, J45 percent of-the marks were vertical:, 
three percent horizontal, and: 39 .percent termed "other." 



Table. 5. Pupil marking direption with three types of response boxes 



Shape-^f 
^_Re;5ponse* Box 


Vertical 




Horizontal 




Square 




Marking Direction 


V H 


C 


0 


V H C 0 


V 


:h C 


0 


Number 


6^83 0 


10^ 


38 


33 374 3- 135 


141 


10 38 


1'24 


Percent 


93 b 


- 4 

1. 


5 ' 


6 67 0, 26 


45 


3 12 


39' 
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Response sheets possessing vertical boxes were read by the 
Digitek 100 DM scaune to deteminer 1) whether .pupils were 
able to mark lin the appropriate positions, and^ 2^ whether pupils 
were able ^p^^ake tnarks: d^fk enough to read by the machine. 
Nineteen errorte were detected in reading 750 marks, an error rate 



of 2. 5 percent A Fourteen :of the errors were due to marks that 
were too light, \reavih|'"five, or 7/10 p: one percent unexplained 
"misreadings** by\ the scanner. Since the horizontal and square 
Response forms\ were not printed to tolerances required by the^ 
scanner, they were-not considered: in this .part of the evaluation* 



\ ObserA^ations recorded by SWRL observers are listed^^ 

. Pupils seemed ^happy^ t o' u se ,penc jils ♦ For many/ of irhara it 
was the first time that they had used: any writing instru- 
ment other than crayons in the classroom. 

---------- - \ c 

\ 

. By the thirds item^the pupils started to mark the appropriate 
answer without specific, instructions from the teacher. That 



is, pnce the appropriate cues were given they did- not need 
to be told to fill in the box* 



10 . ■ > 

V 

VJhen the chyLldrenrwefe reminded after item three to make 
their bor.es dark and shiny many of them went back to items'- 
one and two to'^iriake^sure that the boxes they had filled in 
were shiny. Repetition of »instructidns is helpful in. 

eliciting appropriate behaviors. 

. ' / ^ 

Some teachers felt that there are too tnany rows on the 

test, but that with. tr4ning the pupils could leairn hpw; 

to take the Grit ersion Exercise. They agreed that a practice 
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exercise similar to the practice Criterion Exercise usp 
,in FYCSP would bs' a greaH help in. the tesC-itaking process./ 

' it was noticed that sev^rai of the pupils ,, especially/ the 
younger ones, turned/^e page pnerhaif turn :So that they 
couW f ili in the horizontal boxes by making vertical 
marks. ATsq., several students curled their hands, so 
that when they f iiledi in the: horizontal: boxes' they were 
stiti making a vertical mark. More than one pupil was 
noticed to outline t;he hofizontai box before filling it in. 

One teacher indicated that the marking quality of the pencils 
was not consistent. A ^hcck showed^ that some .pencils coi^ildj' 
hot make dark marks. \ 

Testing time including directions and practice was approx- 
imately 2g to 25 minutes. = 
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VI. Conclusions . 



While the study was pfeliminary , several cautious conclusions can 

be drawn. The mo St -important is the fact the Kindergarten pupils 

appear to be able to handle a ten item per page two-sided Criterion 

Exercise. Observation indicated that them--' ''••culty- is that 

of identifying and maintaining the proper i... .i which to make the 

response. With few exceptions j marks made by pupils were sufficiently 

^y' ■ ' ^ ■ 

dark and complete to be read by the optical scanner. The problem of 

light marks was- traced in same cases to the quality ^of the leaid in;: 

the primary pencils. ' ' . . 

Multiple marks, I.e. , two or more obexes per item filled in, cannot 
'be interpreted for correct response by the machine , Isln^e 'all tJarks 
are read. While a :human may be able to decidip. which response the 
pupil did: intend as: his correct answer and may be able tp retirave the 
inappropriate response by. pre-processing, it is sometimes difficulty 
or impossible to determine -which ^espon^e is; intendedv Assuming that 
the pupil intendedf^o make only onei choice, two possible reasons exist 
for multiple marks : 1 )• the ;pupii did: hot know how to cross out the 
incorrect response (It was noted= in classroom observation that many 
pupils do -not '.enow how or were not able to mark a large X) ; or 2): the 
.pupii-did: not know or remember that he had- to cross out the incorrect 
response.x Stray marks will always be a problem with scannable heets. 
The best resSlt^ition to this situation appears to be reminding the 
pupils not to make"^extranepus marks on- the paper. 




s>vng. responses may have two causes: 1) the pupil could not 



find the proper row on the page in which to fill In his answer; or 
I the pupil- did not -know which answer to fill in after he had 

identified the row. _ , 

The direction in which pupils filled in the response boxes . 
appears to^e guided by the- shape of the box. Over 90 percent of 
the marks in the vertical boxes were vertical and approximately 
two^irds of the horizontal boxes contained horizontal marks. It 
is interesting to note that 45 percent of the pupils filled in the 
square boxes with vertical marks, while only three percent of the 
pupils used a horizontal mark. It does appear , however, that 
puptls could-be taught to use either horizontal marks or vertical 
marks that are dark arid complete- enough; to be read: by. an optical 
scanner. Pupils used square boxes effectively, but the response 
area was so large that filling in the box became almost a coloring: 

exercise. Smaller square boxes wouW seem more acceptable. 

i . • ' ■ - 

1\ 



VII> Suggested Revisions ' 

Th; fotlowing' suggestions should eliminate most of the problems 
encounteredv 1) The initiation of a :practice Criterion Exe^^ j 
/ should familiarize pupils with the ten- it em page and the necessity 

of fMllng in response positions completely with marks that are dark 
and shiny. 2) T^4chers should check during the administration of the 
Criterion Exercises to make sure that pupils are . on the right row and 
that marks are dark arid comp ret e. 3) To help correct the problem of 
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multiple marks teachers should demonstrate how to cross out a filled- 
in box' and remind pupils once or twice during th^ administration of 
the •Criterion Exercise to cross out any unwanted responses. A second- 
possible alternative is ^hat of issuing erasers^ to pupils. In one 
classroom observed, pupils Used erasers with success. 4) Teachers , 
should- remind pupils not to make stray marks on the response sheets, 
and should check during the administering of the Criterion Exercise 
to make sure that pupils are marking response positions only. 



VIII. Furtlier Research - - — _ 

This study broduced encouraging 'answers to the four questions > 
addressed^ However, the results B^st be termed pteliminar^r..^^^^ 
following areas shouW be investigatedr-in depthr 
^ 1/, th^^ffect^of several forms design= on pupil test taking 
behavior-, 

2.. the development ahd= refinement of training procedures that 

produce appropriate teacher and pupil response during assess- 
ment ; 

3. the cost for various types of Criterion Exercisies, including 
the scannable bopklet and test booklet /separate answer sheet 
format; and 

4. the determination of grade levels at which various types of 
assessment devices may be. introduced and used effectively. 

' ' ^ -1 
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